Discrete Recurrent Neural Networks as Pushdown Automata

in this paper we describe a new discrete rccurrcnt neural network model with discrete external stacks for learning context-free grammars (or pushdown automata). Conventional analog recurrent networks tend to have stability problems when presented with input sirings which are longer than those used for training: the network’s internal states become merged and the string can not be correctly parsed. However, the discrete recurrent structure forms a stable representation during learning by using isolated discrete points as its internal representation of states for the automata. Hence, once successfully trained, the network is perfectly stable on input strings of arbitrary length. For training such discrete networks a novel “pseudo-gradient” learning rule is used, Experimental results demonstrate the ability of the discrete network to learn context-free grammars in a stable manner. ‘1’hc discrete network model results in the advantages of a stable network, a clear understanding of the operation of the stack, and a structure which is easily irnplcmcntable in hardware.